Unsupervised Learning Project¶
(Naveen Pete, PGP-AIML Online August 20-A)¶
Data Description¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain¶
Object recognition
Context¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information¶
- All the features are geometric features extracted from the silhouette.
- All are numeric in nature.
Learning Outcomes¶
- Exploratory Data Analysis
- Reduce number dimensions in the dataset with minimal information loss
- Train a model using Principal Components
Objective¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
Import the necessary libraries¶
In [1]:
# Libraries needed for scientific computation, data analysis and manipulation
import numpy as np
import pandas as pd
# Libraries needed for statistical data visualization
from matplotlib import pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
%matplotlib inline
1. Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm (10 marks)¶
Load the dataset¶
In [2]:
# Initialize dataframe with Vehicle dataset
main_df = pd.read_csv("vehicle.csv")
Get a feel of the data¶
In [3]:
main_df.head()
Out[3]:
Count number of rows (records) & columns (attributes or features) in the dataset¶
In [4]:
row_count, column_count = main_df.shape
print(f'No. of rows: {row_count}')
print(f'No. of columns: {column_count}')
Understand the structure of the dataset and the type of attributes within it¶
In [5]:
main_df.info()
Observations
- There are 846 rows and 19 columns in the dataset
- All the columns except 'class' are numerical
- 'class' column contains the class of vehicles. It is the target variable
Check the presence of null and zero(0) values¶
In [6]:
main_df.isnull().apply(pd.value_counts)
Out[6]:
In [7]:
# function to print and return column names that contain null values
def get_columns_with_null_values(df):
print('Columns with null values:')
print('-------------------------')
columns_with_null_values = []
ctr = 0
for col in df.columns:
null_count = df[col].isnull().sum()
if null_count > 0:
columns_with_null_values.append(col)
ctr = ctr + 1
print(f'{ctr}. {col} - {null_count}')
return columns_with_null_values
# function to print and return column names that contain zero(0) values
def get_columns_with_zero_values(df):
print('Columns with zero values:')
print('-------------------------')
columns_with_zero_values = []
ctr = 0
for col in df.columns:
col_type = df[col].dtypes
if col_type != 'object':
zero_count = df[main_df[col]==0][col].count()
if zero_count > 0:
columns_with_zero_values.append(col)
ctr = ctr + 1
print(f'{ctr}. {col} - {zero_count}')
return columns_with_zero_values
# function to impute null values with median value
def impute_null_values(df, columns):
copied_df = df.copy()
for col in columns:
value = copied_df[col].median()
copied_df[col] = copied_df[col].fillna(value)
return copied_df
In [8]:
columns_with_null_values = get_columns_with_null_values(main_df)
columns_with_null_values
Out[8]:
In [9]:
columns_with_zero_values = get_columns_with_zero_values(main_df)
columns_with_zero_values
Out[9]:
Impute null values in the dataset¶
In [10]:
data_df = impute_null_values(main_df, columns_with_null_values)
Before imputing null values¶
In [11]:
plt.figure(figsize=(10, 4))
sns.heatmap(main_df.isnull(), yticklabels=False, cbar=False, cmap='viridis')
Out[11]:
After imputing null values¶
In [12]:
plt.figure(figsize=(10, 4))
sns.heatmap(data_df.isnull(), yticklabels=False, cbar=False, cmap='viridis')
Out[12]:
Observations
- 14 columns had null values. They have been imputed with median values
- 2 columns have zero values. Right now, not handling these columns
2. Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (10 points)¶
In [13]:
data_df.info()
In [14]:
data_df.describe().transpose()
Out[14]:
Separate independent and dependent variables¶
In [15]:
data_df.columns
Out[15]:
In [16]:
# separate independent and dependent variables
independent_features = [
'compactness', 'circularity', 'distance_circularity',
'radius_ratio', 'pr.axis_aspect_ratio', 'max.length_aspect_ratio',
'scatter_ratio', 'elongatedness', 'pr.axis_rectangularity',
'max.length_rectangularity', 'scaled_variance', 'scaled_variance.1',
'scaled_radius_of_gyration', 'scaled_radius_of_gyration.1', 'skewness_about',
'skewness_about.1', 'skewness_about.2', 'hollows_ratio'
]
independent_feature_set_1 = [
'compactness', 'circularity', 'distance_circularity',
'radius_ratio', 'pr.axis_aspect_ratio', 'max.length_aspect_ratio',
'scatter_ratio', 'elongatedness', 'pr.axis_rectangularity',
]
independent_feature_set_2 = [
'max.length_rectangularity', 'scaled_variance', 'scaled_variance.1',
'scaled_radius_of_gyration', 'scaled_radius_of_gyration.1', 'skewness_about',
'skewness_about.1', 'skewness_about.2', 'hollows_ratio'
]
dependent_feature = 'class'
Common functions required for plotting distribution, boxplot, correlation and identifying correlated columns¶
In [17]:
# function to plot the distribution of numeric columns
def plot_distribution(df, columns):
for col in columns:
sns.displot(data=df, kind='hist', x=col, bins=20, kde=True)
# function to plot the boxplot distribution of numeric columns
def plot_boxplot(df, x_column, y_columns):
plt.figure(figsize=(25, 25))
index = 0
for col in y_columns:
index = index + 1
plt.subplot(3, 3, index)
sns.boxplot(x=x_column, y=col, data=df)
# function to plot correlation using a heatmap
def plot_correlation(df):
corr = df.corr()
plt.figure(figsize=(15, 15))
sns.heatmap(corr, annot=True, fmt='.2f')
# function to plot distribution and box plots for a given set of columns
def plot_dist_and_box(df, columns):
for col in columns:
fig, (g1, g2) = plt.subplots(nrows=1, ncols=2)
fig.set_size_inches(15, 3)
sns.histplot(x=df[col], ax=g1, kde=True, bins=20)
g1.set_title(f'Distribution: {col}')
sns.boxplot(x=df[col], ax=g2)
g2.set_title(f'Box Plot: {col}')
# function to return highly correlated columns
def get_corr_features(df, threshold):
corr_columns = set()
corr_matrix = df.corr()
ctr = 0
for i in range(len(corr_matrix.columns)):
for j in range(i):
corr_value = corr_matrix.iloc[i, j]
if corr_value >= threshold:
col_name_i = corr_matrix.columns[i]
col_name_j = corr_matrix.columns[j]
corr_columns.add(col_name_i)
ctr += 1
print(f'{ctr}. \'{col_name_i}\' and \'{col_name_j}\' are highly correlated (Correlation value = {corr_value})')
return corr_columns
# function to identify and treat outlier columns
def treat_outliers(df, columns):
copied_df = df.copy()
for col in columns:
q1 = copied_df[col].quantile(0.25)
q3 = copied_df[col].quantile(0.75)
iqr = q3 - q1
low = q1 - 1.5 * iqr
high = q3 + 1.5 * iqr
value = copied_df[col].median()
copied_df.loc[ (copied_df[col] < low) | (copied_df[col] > high), col] = value
return copied_df
Plot distribution and outliers within independent columns¶
In [18]:
plot_dist_and_box(data_df, independent_features)
Observations
- Summary of observations given below:
- compactness - Approx normal distribution, no outliers
- circularity - Slightly right skewed
- distance_circularity - Left skewed
- radius_ratio - Approx normal distribution, with outliers
- pr.axis_aspect_ratio - Approx normal distribution, with outliers
- max.length_aspect_ratio - 2 peaks, with outliers
- scatter_ratio - 2 peaks, right skewed
- elongatedness - 2 peaks, left skewed
- pr.axis_rectangularity - 2 peaks, right skewed
- max.length_rectangularity - 3 peaks
- scaled_variance - 2 peaks, with outliers
- scaled_variance.1 - 2 peaks, with outliers
- scaled_radius_of_gyration - Slightly right skewed
- scaled_radius_of_gyration.1 - Has multiple outliers
- skewness_about - Right skewed, with outliers
- skewness_about.1 - Has outliers
- skewness_about.2 - No outliers
- hollows_ratio - 2 peaks, no outliers
- Following columns have outliers:
- radius_ratio
- pr.axis_aspect_ratio
- max.length_aspect_ratio
- scaled_variance
- scaled_variance.1
- scaled_radius_of_gyration.1
- skewness_about
- skewness_about.1
In [19]:
# plot_boxplot(data_df, dependent_feature, independent_feature_set_1)
In [20]:
# plot_boxplot(data_df, dependent_feature, independent_feature_set_2)
Verify the distribution after treating outliers¶
In [21]:
columns_with_outliers = [
'radius_ratio', 'pr.axis_aspect_ratio', 'max.length_aspect_ratio', 'scaled_variance',
'scaled_variance.1', 'scaled_radius_of_gyration.1', 'skewness_about', 'skewness_about.1'
]
treated_data_df = treat_outliers(data_df, columns_with_outliers)
plot_dist_and_box(treated_data_df, independent_features)
Target column distribution and observations¶
In [22]:
treated_data_df[dependent_feature].value_counts()
Out[22]:
In [23]:
plot_distribution(treated_data_df, [dependent_feature])
In [24]:
sns.countplot(data=treated_data_df, x=dependent_feature)
Out[24]:
Observations
- 51% of the total rows contain cars
- 26% of the total rows contain buses
- 23% of the total rows contain vans
In [25]:
X = treated_data_df[independent_features]
y = treated_data_df[dependent_feature]
In [26]:
X.shape
Out[26]:
In [27]:
y.shape
Out[27]:
In [28]:
X.head()
Out[28]:
In [29]:
y.head()
Out[29]:
Study the correlation between independent columns¶
In [30]:
plot_correlation(X)
In [31]:
# get all columns that are correlated greater than 90%
highly_corr_features = get_corr_features(X, 0.90)
highly_corr_features
Out[31]:
In [32]:
sns.pairplot(X, diag_kind='kde')
Out[32]:
Observations
- From the above correlation heatmap and pairplot grid, we can observe that there are many columns that have very high collinearity. Hence we can consider using Principal Component Analysis (PCA) for reducing redundant columns
3. Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn) (5 marks)¶
Scale the dataset¶
In [33]:
from sklearn.preprocessing import MinMaxScaler, StandardScaler
# function to scale the dataset
def get_scaled_df(df):
scaler = StandardScaler()
scaler_result = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaler_result, columns=list(df.columns))
return scaled_df
In [34]:
scaled_X = get_scaled_df(X)
scaled_X.head()
Out[34]:
Split data into train and test¶
In [35]:
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=10)
X_train, X_test, y_train, y_test = train_test_split(scaled_X, y, test_size=0.3, random_state=10)
In [36]:
X_train.shape
Out[36]:
In [37]:
y_train.shape
Out[37]:
In [38]:
X_test.shape
Out[38]:
In [39]:
y_test.shape
Out[39]:
4. Train a Support Vector Machine (SVM) using the train set and get the accuracy on the test set (10 marks)¶
In [40]:
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
def init_svc_classifier():
return SVC(gamma='auto')
def train_and_predict(model_name, model, X_train, y_train, X_test, y_test):
print(f'MODEL: {model_name}\n')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
score_train = round(model.score(X_train, y_train), 4) * 100
print(f'Train Accuracy Score: {score_train}')
score_test = round(model.score(X_test, y_test), 4) * 100
print(f'Test Accuracy Score: {score_test}')
model_dict = {
'y_pred': y_pred,
'model_score_train': score_train,
'model_score_test': score_test
}
return model_dict
def get_metrics(model_name, y_test, y_pred):
print('---------------------')
print('Classification Report')
print('---------------------')
cr = classification_report(y_test, y_pred)
print(cr)
print('----------------')
print('Confusion Matrix')
print('----------------')
cm = confusion_matrix(y_test, y_pred)
print(cm)
sns.heatmap(cm, annot=True, fmt='g', cbar=False, cmap='YlOrRd')
accuracy = round(accuracy_score(y_test, y_pred), 4) * 100
f1 = round(f1_score(y_test, y_pred, average='micro'), 4) * 100
precision = round(precision_score(y_test, y_pred, average='micro'), 4) * 100
recall = round(recall_score(y_test, y_pred, average='micro'), 4) * 100
score_dict = {
'accuracy': accuracy,
'f1': f1,
'precision': precision,
'recall': recall
}
return score_dict
Train SVM and print metrics¶
In [41]:
model = init_svc_classifier()
model_dict_svm = train_and_predict('SVM', model, X_train, y_train, X_test, y_test)
score_dict_svm = get_metrics('SVM', y_test, model_dict_svm['y_pred'])
In [42]:
model_dict_svm
Out[42]:
In [43]:
score_dict_svm
Out[43]:
5. Perform K-fold cross validation and get the cross validation score of the model (optional)¶
In [44]:
from sklearn.model_selection import KFold, cross_val_score
def cross_validate(model, X, y):
num_folds = 10
seed = 10
kfold = KFold(n_splits=num_folds, random_state=seed, shuffle=True)
scores = cross_val_score(model, X, y, cv=kfold)
cv_score_dict = {
'mean': round(scores.mean(), 4) * 100,
'std': round(scores.std(), 4) * 100,
'min': round(scores.min(), 4) * 100,
'max': round(scores.max(), 4) * 100
}
print('-------------------------------')
print('Cross Validation Accuracy Score')
print('-------------------------------')
print(f"1. Mean: {cv_score_dict['mean']}")
print(f"2. Std. Dev.: {cv_score_dict['std']}")
print(f"3. Min: {cv_score_dict['min']}")
print(f"4. Max: {cv_score_dict['max']}")
return cv_score_dict
In [45]:
model = init_svc_classifier()
cv_score_svm = cross_validate(model, scaled_X, y)
In [46]:
cv_score_svm
Out[46]:
6. Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data – (10 points)¶
In [47]:
scaled_X.head()
Out[47]:
In [48]:
scaled_X.shape
Out[48]:
Use PCA to identify principal components¶
In [49]:
from sklearn.decomposition import PCA
pca = PCA()
pca.fit(scaled_X)
Out[49]:
In [50]:
# eigen values
pca.explained_variance_
Out[50]:
In [51]:
# eigen vectors
pca.components_
Out[51]:
In [52]:
# variance ratio
pca.explained_variance_ratio_
Out[52]:
In [53]:
# plot eigen values
plt.bar(list(range(1,19)), pca.explained_variance_ratio_, alpha=0.5, align='center')
plt.ylabel('Variation Explained')
plt.xlabel('Eigen Value')
Out[53]:
In [54]:
cumulative_sum = np.cumsum(pca.explained_variance_ratio_)
cumulative_sum
Out[54]:
In [55]:
# plot using step function
plt.step(list(range(1,19)), cumulative_sum, where = 'mid')
plt.ylabel('Cumulative Variation Explained')
plt.xlabel('Eigen Value')
Out[55]:
Observations
- From the above plot, we can observe that the first 7 or 8 components explain 95% variation in the data. Let us consider the first 8 components for building the model
Construct the dataset using PCA components¶
In [56]:
pca = PCA(n_components=8)
pca_result = pca.fit_transform(scaled_X)
pca_X = pd.DataFrame(pca_result)
pca_X.shape
Out[56]:
In [57]:
y.shape
Out[57]:
Study the correlation between PCA components¶
In [58]:
plot_correlation(pca_X)
In [59]:
sns.pairplot(pca_X, diag_kind='kde')
Out[59]:
Observations
- From the above correlation heatmap and pairplot grid, we can now observe that the columns have no collinearity after we have used Principal Component Analysis (PCA) to reduce dimensions
7. Repeat steps 3, 4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state) (20 marks)¶
7.1. Split the data into train and test¶
In [60]:
pca_X_train, pca_X_test, pca_y_train, pca_y_test = train_test_split(pca_X, y, test_size=0.3, random_state=10)
In [61]:
pca_X_train.shape
Out[61]:
In [62]:
pca_y_train.shape
Out[62]:
In [63]:
pca_X_test.shape
Out[63]:
In [64]:
pca_y_test.shape
Out[64]:
7.2. Train a Support Vector Machine (SVM) using the train set and get the accuracy on the test set¶
In [65]:
model = init_svc_classifier()
model_dict_svm_pca = train_and_predict('SVM with PCA', model, pca_X_train, pca_y_train, pca_X_test, pca_y_test)
score_dict_svm_pca = get_metrics('SVM with PCA', pca_y_test, model_dict_svm_pca['y_pred'])
In [66]:
model_dict_svm_pca
Out[66]:
In [67]:
score_dict_svm_pca
Out[67]:
7.3. Perform K-fold cross validation and get the cross validation score of the model (optional)¶
In [68]:
model = init_svc_classifier()
cv_score_svm_pca = cross_validate(model, pca_X, y)
In [69]:
cv_score_svm_pca
Out[69]:
8. Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)¶
In [70]:
accuracy_data = [
[
'SVM - All', model_dict_svm['model_score_train'], model_dict_svm['model_score_test'],
score_dict_svm['accuracy'], score_dict_svm['f1'], score_dict_svm['precision'], score_dict_svm['recall']
],
[
'SVM - PCA', model_dict_svm_pca['model_score_train'], model_dict_svm_pca['model_score_test'],
score_dict_svm_pca['accuracy'], score_dict_svm_pca['f1'], score_dict_svm_pca['precision'], score_dict_svm_pca['recall']
]
]
accuracy_score_df = pd.DataFrame(
accuracy_data, columns = ['Model', 'Train Score', 'Test Score', 'Accuracy', 'F1', 'Precision', 'Recall']
)
In [71]:
cv_data = [
[ 'SVM - All', cv_score_svm['mean'], cv_score_svm['std'], cv_score_svm['min'], cv_score_svm['max'] ],
[ 'SVM - PCA', cv_score_svm_pca['mean'], cv_score_svm_pca['std'], cv_score_svm_pca['min'], cv_score_svm_pca['max'] ]
]
cv_score_df = pd.DataFrame(
cv_data, columns = ['Model', 'Mean', 'Std. Dev.', 'Min', 'Max']
)
Accuracy Score Result¶
In [72]:
accuracy_score_df
Out[72]:
Cross Validation Score Result¶
In [73]:
cv_score_df
Out[73]:
Observations & Conclusion
- Above results clearly show that, by using PCA we are able to reduce the dimensions from 18 features to 8 features. However there is a slight dip in both accuracy and cross validation score results.